This is a guest authored post by Yu Chen, Senior Software Engineer, Headspace. Headspace's core products are iOS, Android and web-based apps that focus on improving the health and happiness of its users through mindfulness, meditation, sleep, exercise and focus content. Machine learning (ML) models are core to our user experiences by offering recommendations that engage users with new relevant, personalized content that builds consistent habits in their lifelong journey. Data fed to ML models is often most valuable when it can be immediately leveraged to make decisions in the moment, but, traditionally, consumer data is ingested, transformed, persisted and sits dormant for lengthy periods of time before machine learning and data analytics teams leverage it. Finding a way to leverage user data to generate real-time insights and decisions means that consumer-facing products like the Headspace app can dramatically shorten the end-to-end user feedback loop: actions that users perform just moments prior can be incorporated into the product to generate more relevant, personalized and context-specific content recommendation for the user.

A Data Model for Training and Scoring 5 Members Events Event RSVPs Model Member Predicted RSVP Offline Training Real-Time Scoring 6. Component Integration and Serving 6 Kinesis Producer AWS S3 Spark Model Training Spark Structured Streaming Meetup Stream Meetup Member API Meetup Prediction 7. Producing the Kinesis Firehose Stream 7 requests.get() Save the model to disk for scoringmodel.write.overwrite().save(...) 10. Scoring the Model in Real-time 10 Load the trained modelval model PipelineModel.load(...) Stream meetup event data Score the model val events spark.readStream ML Limitations in Structured Streaming 11 •Structured streaming does not support operations needed by ML methods –count, collect, round, aggregate*, etc. • Many models, transformers, and estimators are not supported –K-Means, SVM, CountVectorizer, VectorAssembler, StringIndexer, etc. 12.

Ground level definitions out of the way, what has Databricks been doing to add to unified utopia? The company has this month announced Apache Spark open-source cluster-computing framework. This means that the company is the vendor to support Apache Spark 2.3 within a compute engine, Databricks Runtime 4.0, which is now generally available. In addition to support for Spark 2.3, Databricks Runtime 4.0 introduces new features including Machine Learning Model Export to simplify production deployments and performance optimizations. "The community continues to expand on Apache Spark's role as a unified analytics engine for big data and AI. This is a major milestone to introduce the continuous processing mode of Structured Streaming with millisecond low-latency, as well as other features across the project," said Matei Zaharia, creator of Apache Spark and chief technologist and co-founder of Databricks.

From its humble beginnings in the AMPLab at U.C. Berkeley in 2009, Apache Spark has become one of the key big data distributed processing frameworks in the world. Spark can be deployed in a variety of ways, provides native bindings for the Java, Scala, Python, and R programming languages, and supports SQL, streaming data, machine learning, and graph processing. You'll find it used by banks, telecommunications companies, games companies, governments, and all of the major tech giants such as Apple, Facebook, IBM, and Microsoft. Out of the box, Spark can run in a standalone cluster mode that simply requires the Apache Spark framework and a JVM on each machine in your cluster. However, it's more likely you'll want to take advantage of a resource or cluster management system to take care of allocating workers on demand for you.

Want to watch this again later? Sign in to add this video to a playlist. Report Need to report the video? Sign in to report inappropriate content. Report Need to report the video?

Since the creation of Apache Spark, I/O throughput has increased at a faster pace than processing speed. In a lot of big data applications, the bottleneck is increasingly the CPU. With the release of Apache Spark 2.0 and Project Tungsten, Spark runs a number of control operations close to the metal. At the same time, there has been a surge of interest in using GPUs (the Graphics Processing Units of video cards) for general purpose applications, and a number of frameworks have been proposed to do numerical computations on GPUs. In this talk, we will discuss how to combine Apache Spark with TensorFlow, a new framework from Google that provides building blocks for Machine Learning computations on GPUs.

To learn more about Spark's Machine Learning APIs, check out Holden Karau's and Seth Hendrickson's session Extending Spark ML at Spark Summit West 2017 on Tuesday, June 6 2:00 PM (Room 2). Spark's new ALPHA Structured Streaming API has caused a lot of excitement because it brings the Data set/DataFrame/SQL APIs into a streaming context. In this initial version of Structured Streaming, the machine learning APIs have not yet been integrated. However, this doesn't stop us from having fun exploring how to get machine learning to work with Structured Streaming. For our Spark Structured Streaming for machine learning talk on at Strata Hadoop World New York 2016, we've started early proof-of-concept work to integrate structured streaming and machine learning available in the spark-structured-streaming-ml repo.

To learn more about Structured Streaming and Machine Learning, check out Holden Karau's and Seth Hendrickson's session Spark Structured Streaming for machine learning at Strata Hadoop World New York, September 26-29, 2016. Spark's new ALPHA Structured Streaming API has caused a lot of excitement because it brings the Data set/DataFrame/SQL APIs into a streaming context. In this initial version of Structured Streaming, the machine learning APIs have not yet been integrated. However, this doesn't stop us from having fun exploring how to get machine learning to work with Structured Streaming. For our Spark Structured Streaming for machine learning talk on at Strata Hadoop World New York 2016, we've started early proof-of-concept work to integrate structured streaming and machine learning available in the spark-structured-streaming-ml repo.

Cognitive business took a bold leap forward in New York City the week of 26 September 2016. At two events on Manhattan's west side, IBM led customers, partners and industry at large in an exploration of how to put machine learning, artificial intelligence (AI), and big data analytics to work. At the extremely well-attended IBM DataFirst Launch Event at Hudson Mercantile, the chief news was the announcement of Project DataWorks. This new, cloud-based offering provides a self-service environment for teams of data scientists, data engineers and other professionals to collaboratively develop, iterate and deploy sophisticated AI, cognitive computing, machine learning and other advanced analytics. Check out what Dinesh Nirmal, vice president, IBM Analytics, had to say about DataWorks in action during the Strata Hadoop World 2016 conference.